

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />

  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>Testing changes to the Linux Kernel CephFS driver &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/css/custom.css" type="text/css" />

  
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/jquery.js"></script>
        <script src="../../_static/_sphinx_javascript_frameworks_compat.js"></script>
        <script data-url_root="../../" id="documentation_options" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/doctools.js"></script>
        <script src="../../_static/sphinx_highlight.js"></script>
    
    <script type="text/javascript" src="../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../genindex/" />
    <link rel="search" title="Search" href="../../search/" />
    <link rel="next" title="Hacking on Ceph in Kubernetes with Rook" href="../kubernetes/" />
    <link rel="prev" title="IANA 号" href="../iana/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../../" class="icon icon-home" aria-label="Home"></a></li>
          <li class="breadcrumb-item"><a href="../internals/">Ceph 内幕</a></li>
      <li class="breadcrumb-item active">Testing changes to the Linux Kernel CephFS driver</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../../_sources/dev/kclient.rst.txt" rel="nofollow"> View page source</a>
      </li>
  </ul>
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../" class="icon icon-home"> Ceph
          

          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../start/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../cephadm/">Cephadm</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../monitoring/">监控概览</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../developer_guide/">开发者指南</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../internals/">Ceph 内幕</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../balancer-design/">Ceph 如何均衡（读写、容量）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../blkin/">Tracing Ceph With LTTng</a></li>
<li class="toctree-l2"><a class="reference internal" href="../blkin/#tracing-ceph-with-blkin">Tracing Ceph With Blkin</a></li>
<li class="toctree-l2"><a class="reference internal" href="../bluestore/">BlueStore Internals</a></li>
<li class="toctree-l2"><a class="reference internal" href="../ceph_krb_auth/">如何配置好 Ceph Kerberos 认证的详细文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephfs-mirroring/">CephFS Mirroring</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephfs-reclaim/">CephFS Reclaim Interface</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephfs-snapshots/">CephFS 快照</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephx/">Cephx</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephx_protocol/">Cephx 认证协议详细阐述</a></li>
<li class="toctree-l2"><a class="reference internal" href="../config/">配置管理系统</a></li>
<li class="toctree-l2"><a class="reference internal" href="../config-key/">config-key layout</a></li>
<li class="toctree-l2"><a class="reference internal" href="../context/">CephContext</a></li>
<li class="toctree-l2"><a class="reference internal" href="../continuous-integration/">Continuous Integration Architecture</a></li>
<li class="toctree-l2"><a class="reference internal" href="../corpus/">资料库结构</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cpu-profiler/">Oprofile 的安装</a></li>
<li class="toctree-l2"><a class="reference internal" href="../crush-msr/">CRUSH MSR (Multi-step Retry)</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cxx/">C++17 and libstdc++ ABI</a></li>
<li class="toctree-l2"><a class="reference internal" href="../deduplication/">去重</a></li>
<li class="toctree-l2"><a class="reference internal" href="../delayed-delete/">CephFS delayed deletion</a></li>
<li class="toctree-l2"><a class="reference internal" href="../dev_cluster_deployment/">开发集群的部署</a></li>
<li class="toctree-l2"><a class="reference internal" href="../dev_cluster_deployment/#id5">在同一机器上部署多套开发集群</a></li>
<li class="toctree-l2"><a class="reference internal" href="../development-workflow/">开发流程</a></li>
<li class="toctree-l2"><a class="reference internal" href="../documenting/">为 Ceph 写作文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../dpdk/">Ceph messenger DPDKStack</a></li>
<li class="toctree-l2"><a class="reference internal" href="../encoding/">序列化（编码、解码）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../erasure-coded-pool/">纠删码存储池</a></li>
<li class="toctree-l2"><a class="reference internal" href="../file-striping/">File striping</a></li>
<li class="toctree-l2"><a class="reference internal" href="../freebsd/">FreeBSD Implementation details</a></li>
<li class="toctree-l2"><a class="reference internal" href="../generatedocs/">Ceph 文档的构建</a></li>
<li class="toctree-l2"><a class="reference internal" href="../health-reports/">Health Reports</a></li>
<li class="toctree-l2"><a class="reference internal" href="../iana/">IANA 号</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">Testing changes to the Linux Kernel CephFS driver</a></li>
<li class="toctree-l2"><a class="reference internal" href="#step-one-build-the-kernel">Step One: build the kernel</a></li>
<li class="toctree-l2"><a class="reference internal" href="#step-two-create-a-vm">Step Two: create a VM</a></li>
<li class="toctree-l2"><a class="reference internal" href="#step-three-networking-the-vm">Step Three: Networking the VM</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#step-four-mounting-a-cephfs-file-system-in-your-vm">Step Four: mounting a CephFS file system in your VM</a></li>
<li class="toctree-l3"><a class="reference internal" href="#step-five-testing-kernel-changes-in-teuthology">Step Five: testing kernel changes in teuthology</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../kubernetes/">Hacking on Ceph in Kubernetes with Rook</a></li>
<li class="toctree-l2"><a class="reference internal" href="../libcephfs_proxy/">Design of the libcephfs proxy</a></li>
<li class="toctree-l2"><a class="reference internal" href="../libs/">库体系结构</a></li>
<li class="toctree-l2"><a class="reference internal" href="../logging/">集群日志的用法</a></li>
<li class="toctree-l2"><a class="reference internal" href="../logs/">调试日志</a></li>
<li class="toctree-l2"><a class="reference internal" href="../macos/">在 MacOS 上构建</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mempool_accounting/">What is a mempool?</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mempool_accounting/#some-common-mempools-that-we-can-track">Some common mempools that we can track</a></li>
<li class="toctree-l2"><a class="reference internal" href="../messenger/">Messenger notes</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-bootstrap/">Monitor bootstrap</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-elections/">Monitor Elections</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-on-disk-formats/">ON-DISK FORMAT</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-osdmap-prune/">FULL OSDMAP VERSION PRUNING</a></li>
<li class="toctree-l2"><a class="reference internal" href="../msgr2/">msgr2 协议（ msgr2.0 和 msgr2.1 ）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../network-encoding/">Network Encoding</a></li>
<li class="toctree-l2"><a class="reference internal" href="../network-protocol/">网络协议</a></li>
<li class="toctree-l2"><a class="reference internal" href="../object-store/">对象存储架构概述</a></li>
<li class="toctree-l2"><a class="reference internal" href="../osd-class-path/">OSD class path issues</a></li>
<li class="toctree-l2"><a class="reference internal" href="../peering/">互联</a></li>
<li class="toctree-l2"><a class="reference internal" href="../perf/">Using perf</a></li>
<li class="toctree-l2"><a class="reference internal" href="../perf_counters/">性能计数器</a></li>
<li class="toctree-l2"><a class="reference internal" href="../perf_histograms/">Perf histograms</a></li>
<li class="toctree-l2"><a class="reference internal" href="../placement-group/">PG （归置组）说明</a></li>
<li class="toctree-l2"><a class="reference internal" href="../quick_guide/">开发者指南（快速）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rados-client-protocol/">RADOS 客户端协议</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rbd-diff/">RBD 增量备份</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rbd-export/">RBD Export &amp; Import</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rbd-layering/">RBD Layering</a></li>
<li class="toctree-l2"><a class="reference internal" href="../release-checklists/">Release checklists</a></li>
<li class="toctree-l2"><a class="reference internal" href="../release-process/">Ceph Release Process</a></li>
<li class="toctree-l2"><a class="reference internal" href="../seastore/">SeaStore</a></li>
<li class="toctree-l2"><a class="reference internal" href="../sepia/">Sepia 社区测试实验室</a></li>
<li class="toctree-l2"><a class="reference internal" href="../session_authentication/">Session Authentication for the Cephx Protocol</a></li>
<li class="toctree-l2"><a class="reference internal" href="../testing/">测试笔记</a></li>
<li class="toctree-l2"><a class="reference internal" href="../versions/">Public OSD Version</a></li>
<li class="toctree-l2"><a class="reference internal" href="../vstart-ganesha/">NFS CephFS-RGW Developer Guide</a></li>
<li class="toctree-l2"><a class="reference internal" href="../wireshark/">Wireshark Dissector</a></li>
<li class="toctree-l2"><a class="reference internal" href="../zoned-storage/">Zoned Storage Support</a></li>
<li class="toctree-l2"><a class="reference internal" href="../osd_internals/">OSD 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mds_internals/">MDS 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../radosgw/">RADOS 网关开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../ceph-volume/">ceph-volume 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../crimson/">Crimson developer documentation</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../hardware-monitoring/">硬件监控</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <section id="testing-changes-to-the-linux-kernel-cephfs-driver">
<h1>Testing changes to the Linux Kernel CephFS driver<a class="headerlink" href="#testing-changes-to-the-linux-kernel-cephfs-driver" title="Permalink to this heading"></a></h1>
<p>This walkthrough will explain one (opinionated) way to do testing of the Linux
kernel client against a development cluster. We will try to mimimize any
assumptions about pre-existing knowledge of how to do kernel builds or any
related best-practices.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>There are many completely valid ways to do kernel development for
Ceph. This guide is a walkthrough of the author’s own environment.
You may decide to do things very differently.</p>
</div>
</section>
<section id="step-one-build-the-kernel">
<h1>Step One: build the kernel<a class="headerlink" href="#step-one-build-the-kernel" title="Permalink to this heading"></a></h1>
<p>Clone the kernel:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>git<span class="w"> </span>init<span class="w"> </span>linux<span class="w"> </span><span class="o">&amp;&amp;</span><span class="w"> </span><span class="nb">cd</span><span class="w"> </span>linux
git<span class="w"> </span>remote<span class="w"> </span>add<span class="w"> </span>torvalds<span class="w"> </span>git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
git<span class="w"> </span>remote<span class="w"> </span>add<span class="w"> </span>ceph<span class="w"> </span>https://github.com/ceph/ceph-client.git
git<span class="w"> </span>fetch<span class="w"> </span><span class="o">&amp;&amp;</span><span class="w"> </span>git<span class="w"> </span>checkout<span class="w"> </span>torvalds/master
</pre></div>
</div>
<p>Configure the kernel:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>make<span class="w"> </span>defconfig
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>You can alternatively use the <a class="reference external" href="https://github.com/ceph/ceph-build/tree/899d0848a0f487f7e4cee773556aaf9529b8db26/kernel/build">Ceph Kernel QA Config</a> for building the kernel.</p>
</div>
<p>We now have a kernel config with reasonable defaults for the architecture you’re
building on. The next thing to do is to enable configs which will build Ceph and/or
provide functionality we need to do testing.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>cat<span class="w"> </span>&gt;<span class="w"> </span>~/.ceph.config<span class="w"> </span><span class="s">&lt;&lt;EOF</span>
<span class="s">CONFIG_CEPH_FS=y</span>
<span class="s">CONFIG_CEPH_FSCACHE=y</span>
<span class="s">CONFIG_CEPH_FS_POSIX_ACL=y</span>
<span class="s">CONFIG_CEPH_FS_SECURITY_LABEL=y</span>
<span class="s">CONFIG_CEPH_LIB_PRETTYDEBUG=y</span>
<span class="s">CONFIG_DYNAMIC_DEBUG=y</span>
<span class="s">CONFIG_DYNAMIC_DEBUG_CORE=y</span>
<span class="s">CONFIG_FRAME_POINTER=y</span>
<span class="s">CONFIG_FSCACHE</span>
<span class="s">CONFIG_FSCACHE_STATS</span>
<span class="s">CONFIG_FS_ENCRYPTION=y</span>
<span class="s">CONFIG_FS_ENCRYPTION_ALGS=y</span>
<span class="s">CONFIG_KGDB=y</span>
<span class="s">CONFIG_KGDB_SERIAL_CONSOLE=y</span>
<span class="s">CONFIG_XFS_FS=y</span>
<span class="s">EOF</span>
</pre></div>
</div>
<p>Beyond enabling Ceph-related configs, we are also enabling some useful
debug configs and XFS (as an alternative to ext4 if needed for our root file
system).</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>It is a good idea to not build anything as a kernel module. Otherwise, you would need to <code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">modules_install</span></code> on the root drive of the VM.</p>
</div>
<p>Now, merge the configs.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>scripts/kconfig/merge_config.sh<span class="w"> </span>.config<span class="w"> </span>~/.ceph.config
</pre></div>
</div>
<p>Finally, build the kernel:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>make<span class="w"> </span>-j
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>This document does not discuss how to get relevant utilities for your
distribution to actually build the kernel, like gcc. Please use your search
engine of choice to learn how to do that.</p>
</div>
</section>
<section id="step-two-create-a-vm">
<h1>Step Two: create a VM<a class="headerlink" href="#step-two-create-a-vm" title="Permalink to this heading"></a></h1>
<p>A virtual machine is a good choice for testing the kernel client for a few reasons:</p>
<ul class="simple">
<li><p>You can more easily monitor and configure networking for the VM.</p></li>
<li><p>You can very rapidly test a change to the kernel (build -&gt; mount in less than 10 seconds).</p></li>
<li><p>A fault in the kernel won’t crash your machine.</p></li>
<li><p>You have a suite of tools available for analysis on the running kernel.</p></li>
</ul>
<p>The main decision for you to make is what Linux distribution you want to use.
This document uses Arch Linux due to the author’s familiarity. We also use LVM
to create a volume. You may use partitions or whatever mechanism you like to
create a block device. In general, this block device will be used repeatedly in
testing. You may want to use snapshots to avoid a VM somehow corrupting your
root disk and forcing you to start over.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># create a volume</span>
<span class="nv">VOLUME_GROUP</span><span class="o">=</span>foo
sudo<span class="w"> </span>lvcreate<span class="w"> </span>-L<span class="w"> </span>256G<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$VOLUME_GROUP</span><span class="s2">&quot;</span><span class="w"> </span>-n<span class="w"> </span><span class="k">$(</span>whoami<span class="k">)</span>-vm-0
<span class="nv">DEV</span><span class="o">=</span><span class="s2">&quot;/dev/</span><span class="si">${</span><span class="nv">VOLUME_GROUP</span><span class="si">}</span><span class="s2">/</span><span class="k">$(</span>whoami<span class="k">)</span><span class="s2">-vm-0&quot;</span>
sudo<span class="w"> </span>mkfs.xfs<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$DEV</span><span class="s2">&quot;</span>
sudo<span class="w"> </span>mount<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$DEV</span><span class="s2">&quot;</span><span class="w"> </span>/mnt
sudo<span class="w"> </span>pacstrap<span class="w"> </span>/mnt<span class="w"> </span>base<span class="w"> </span>base-devel<span class="w"> </span>vim<span class="w"> </span>less<span class="w"> </span>jq
sudo<span class="w"> </span>arch-chroot<span class="w"> </span>/mnt
<span class="c1"># # delete root&#39;s password for ease of login</span>
<span class="c1"># passwd -d root</span>
<span class="c1"># mkdir -p /root/.ssh &amp;&amp; echo &quot;$YOUR_SSH_KEY_PUBKEY&quot; &gt;&gt; /root/.ssh/authorized_keys</span>
<span class="c1"># exit</span>
sudo<span class="w"> </span>umount<span class="w"> </span>/mnt
</pre></div>
</div>
<p>Once that’s done, we should be able to run a VM:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>qemu-system-x86_64<span class="w"> </span>-enable-kvm<span class="w"> </span>-kernel<span class="w"> </span><span class="k">$(</span><span class="nb">pwd</span><span class="k">)</span>/arch/x86/boot/bzImage<span class="w"> </span>-drive<span class="w"> </span><span class="nv">file</span><span class="o">=</span><span class="s2">&quot;</span><span class="nv">$DEV</span><span class="s2">&quot;</span>,if<span class="o">=</span>virtio,format<span class="o">=</span>raw<span class="w"> </span>-append<span class="w"> </span><span class="s1">&#39;root=/dev/vda rw&#39;</span>
</pre></div>
</div>
<p>You should see output like:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">VNC</span> <span class="n">server</span> <span class="n">running</span> <span class="n">on</span> <span class="p">::</span><span class="mi">1</span><span class="p">:</span><span class="mi">5900</span>
</pre></div>
</div>
<p>You could view that console using:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>vncviewer<span class="w"> </span><span class="m">127</span>.0.0.1:5900
</pre></div>
</div>
<p>Congratulations, you have a VM running the kernel that you just built.</p>
</section>
<section id="step-three-networking-the-vm">
<h1>Step Three: Networking the VM<a class="headerlink" href="#step-three-networking-the-vm" title="Permalink to this heading"></a></h1>
<p>This is the “hard part” and requires the most customization depending on what
you want to do. For this author, I currently have a development setup like:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>  <span class="n">sepian</span> <span class="n">netns</span>
 <span class="n">______________</span>
<span class="o">|</span>              <span class="o">|</span>
<span class="o">|</span> <span class="n">kernel</span> <span class="n">VM</span>    <span class="o">|</span>              <span class="n">sepia</span><span class="o">-</span><span class="n">bounce</span> <span class="n">VM</span>      <span class="n">vossi04</span><span class="o">.</span><span class="n">front</span><span class="o">.</span><span class="n">sepia</span><span class="o">.</span><span class="n">ceph</span><span class="o">.</span><span class="n">com</span>
<span class="o">|</span>  <span class="o">-------</span>  <span class="o">|</span>  <span class="o">|</span>                  <span class="o">------</span>                    <span class="o">-------</span>
<span class="o">|</span>  <span class="o">|</span>     <span class="o">|</span>  <span class="o">|</span>  <span class="o">|</span> <span class="mf">192.168.20.1</span>     <span class="o">|</span>    <span class="o">|</span>                    <span class="o">|</span>     <span class="o">|</span>
<span class="o">|</span>  <span class="o">|</span>     <span class="o">|--|--|-</span> <span class="o">&lt;-</span> <span class="n">wireguard</span> <span class="o">-&gt;</span> <span class="o">|</span>    <span class="o">|</span>  <span class="o">&lt;--</span> <span class="n">sepia</span> <span class="n">vpn</span> <span class="o">-&gt;</span>  <span class="o">|</span>     <span class="o">|</span>
<span class="o">|</span>  <span class="o">|</span><span class="n">_____</span><span class="o">|</span>  <span class="o">|</span>  <span class="o">|</span>     <span class="mf">192.168.20.2</span> <span class="o">|</span><span class="n">____</span><span class="o">|</span>                    <span class="o">|</span><span class="n">_____</span><span class="o">|</span>
<span class="o">|</span>          <span class="n">br0</span> <span class="o">|</span>
<span class="o">|</span><span class="n">______________</span><span class="o">|</span>
</pre></div>
</div>
<p>The sepia-bounce VM is used as a bounce box to the sepia lab. It can proxy ssh
connections, route any sepia-bound traffic, or serve as a DNS proxy. The use of
a sepia-bounce VM is optional but can be useful, especially if you want to
create numerous kernel VMs for testing.</p>
<p>I like to use the vossi04 <a class="reference external" href="https://wiki.sepia.ceph.com/doku.php?id=devplayground#developer_playgrounds">developer playground</a> to build Ceph and setup a
vstart cluster.  It has sufficient resources to make building Ceph very fast
(~5 minutes cold build) and local disk resources to run a decent vstart
cluster.</p>
<p>To avoid overcomplicating this document with the details of the sepia-bounce
VM, I will note the following main configurations used for the purpose of
testing the kernel:</p>
<ul class="simple">
<li><p>setup a wireguard tunnel between the machine creating kernel VMs and the sepia-bounce VM</p></li>
<li><p>use <code class="docutils literal notranslate"><span class="pre">systemd-resolved</span></code> as a DNS resolver and listen on 192.168.20.2 (instead of just localhost)</p></li>
<li><p>connect to the sepia <a class="reference external" href="https://wiki.sepia.ceph.com/doku.php?id=vpnaccess">VPN</a> and use <a class="reference external" href="systemd-resolved:https://wiki.archlinux.org/title/Systemd-resolved">systemd resolved update script</a> to configure <code class="docutils literal notranslate"><span class="pre">systemd-resolved</span></code> to use the DNS servers acquired via DHCP from the sepia VPN</p></li>
<li><p>configure <code class="docutils literal notranslate"><span class="pre">firewalld</span></code> to allow wireguard traffic and to masquerade and forward traffic to the sepia vpn</p></li>
</ul>
<p>The next task is to connect the kernel VM to the sepia-bounce VM. A network
namespace can be useful for this purpose to isolate traffic / routing rules for
the VMs. For me, I orchestrate this using a custom systemd one-shot unit that
looks like:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># create the net namespace</span>
<span class="n">ExecStart</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">ip</span> <span class="n">netns</span> <span class="n">add</span> <span class="n">sepian</span>
<span class="c1"># bring lo up</span>
<span class="n">ExecStart</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">ip</span> <span class="n">netns</span> <span class="n">exec</span> <span class="n">sepian</span> <span class="n">ip</span> <span class="n">link</span> <span class="nb">set</span> <span class="n">dev</span> <span class="n">lo</span> <span class="n">up</span>
<span class="c1"># setup wireguard to sepia-bounce</span>
<span class="n">ExecStart</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">ip</span> <span class="n">link</span> <span class="n">add</span> <span class="n">wg</span><span class="o">-</span><span class="n">sepian</span> <span class="nb">type</span> <span class="n">wireguard</span>
<span class="n">ExecStart</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">wg</span> <span class="n">setconf</span> <span class="n">wg</span><span class="o">-</span><span class="n">sepian</span> <span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">wireguard</span><span class="o">/</span><span class="n">wg</span><span class="o">-</span><span class="n">sepian</span><span class="o">.</span><span class="n">conf</span>
<span class="c1"># move the wireguard interface to the sepian nents</span>
<span class="n">ExecStart</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">ip</span> <span class="n">link</span> <span class="nb">set</span> <span class="n">wg</span><span class="o">-</span><span class="n">sepian</span> <span class="n">netns</span> <span class="n">sepian</span>
<span class="c1"># configure the static ip and bring it up</span>
<span class="n">ExecStart</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">ip</span> <span class="n">netns</span> <span class="n">exec</span> <span class="n">sepian</span> <span class="n">ip</span> <span class="n">addr</span> <span class="n">add</span> <span class="mf">192.168.20.1</span><span class="o">/</span><span class="mi">24</span> <span class="n">dev</span> <span class="n">wg</span><span class="o">-</span><span class="n">sepian</span>
<span class="n">ExecStart</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">ip</span> <span class="n">netns</span> <span class="n">exec</span> <span class="n">sepian</span> <span class="n">ip</span> <span class="n">link</span> <span class="nb">set</span> <span class="n">wg</span><span class="o">-</span><span class="n">sepian</span> <span class="n">up</span>
<span class="c1"># logging info</span>
<span class="n">ExecStart</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">ip</span> <span class="n">netns</span> <span class="n">exec</span> <span class="n">sepian</span> <span class="n">ip</span> <span class="n">addr</span>
<span class="n">ExecStart</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">ip</span> <span class="n">netns</span> <span class="n">exec</span> <span class="n">sepian</span> <span class="n">ip</span> <span class="n">route</span>
<span class="c1"># make wireguard the default route</span>
<span class="n">ExecStart</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">ip</span> <span class="n">netns</span> <span class="n">exec</span> <span class="n">sepian</span> <span class="n">ip</span> <span class="n">route</span> <span class="n">add</span> <span class="n">default</span> <span class="n">via</span> <span class="mf">192.168.20.2</span> <span class="n">dev</span> <span class="n">wg</span><span class="o">-</span><span class="n">sepian</span>
<span class="c1"># more logging</span>
<span class="n">ExecStart</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">ip</span> <span class="n">netns</span> <span class="n">exec</span> <span class="n">sepian</span> <span class="n">ip</span> <span class="n">route</span>
<span class="c1"># add a bridge interface for VMs</span>
<span class="n">ExecStart</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">ip</span> <span class="n">netns</span> <span class="n">exec</span> <span class="n">sepian</span> <span class="n">ip</span> <span class="n">link</span> <span class="n">add</span> <span class="n">name</span> <span class="n">br0</span> <span class="nb">type</span> <span class="n">bridge</span>
<span class="c1"># configure the addresses and bring it up</span>
<span class="n">ExecStart</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">ip</span> <span class="n">netns</span> <span class="n">exec</span> <span class="n">sepian</span> <span class="n">ip</span> <span class="n">addr</span> <span class="n">add</span> <span class="mf">192.168.0.1</span><span class="o">/</span><span class="mi">24</span> <span class="n">dev</span> <span class="n">br0</span>
<span class="n">ExecStart</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">ip</span> <span class="n">netns</span> <span class="n">exec</span> <span class="n">sepian</span> <span class="n">ip</span> <span class="n">link</span> <span class="nb">set</span> <span class="n">br0</span> <span class="n">up</span>
<span class="c1"># masquerade/forward traffic to sepia-bounce</span>
<span class="n">ExecStart</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">ip</span> <span class="n">netns</span> <span class="n">exec</span> <span class="n">sepian</span> <span class="n">iptables</span> <span class="o">-</span><span class="n">t</span> <span class="n">nat</span> <span class="o">-</span><span class="n">A</span> <span class="n">POSTROUTING</span> <span class="o">-</span><span class="n">o</span> <span class="n">wg</span><span class="o">-</span><span class="n">sepian</span> <span class="o">-</span><span class="n">j</span> <span class="n">MASQUERADE</span>
</pre></div>
</div>
<p>When using the network namespace, we will use <code class="docutils literal notranslate"><span class="pre">ip</span> <span class="pre">netns</span> <span class="pre">exec</span></code>. There is a
handy feature to automatically bind mount files into the <code class="docutils literal notranslate"><span class="pre">/etc</span></code> namespace for
commands run via that command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># cat /etc/netns/sepian/resolv.conf</span>
<span class="n">nameserver</span> <span class="mf">192.168.20.2</span>
</pre></div>
</div>
<p>That file will configure the libc name resolution stack to route DNS requests
for applications to the <code class="docutils literal notranslate"><span class="pre">systemd-resolved</span></code> daemon running on sepia-bounce.
Consequently, any application running in that netns will be able to resolve
sepia hostnames:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ sudo ip netns exec sepian host vossi04.front.sepia.ceph.com
vossi04.front.sepia.ceph.com has address 172.21.10.4
</pre></div>
</div>
<p>Okay, great. We have a network namespace that forwards traffic to the sepia
VPN.  The next mental step is to connect virtual machines running a kernel to
the bridge we have configured. The straightforward way to do that is to create
a “tap” device which connects to the bridge:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>sudo<span class="w"> </span>ip<span class="w"> </span>netns<span class="w"> </span><span class="nb">exec</span><span class="w"> </span>sepian<span class="w"> </span>qemu-system-x86_64<span class="w"> </span><span class="se">\</span>
<span class="w">    </span>-enable-kvm<span class="w"> </span><span class="se">\</span>
<span class="w">    </span>-kernel<span class="w"> </span><span class="k">$(</span><span class="nb">pwd</span><span class="k">)</span>/arch/x86/boot/bzImage<span class="w"> </span><span class="se">\</span>
<span class="w">    </span>-drive<span class="w"> </span><span class="nv">file</span><span class="o">=</span><span class="s2">&quot;</span><span class="nv">$DEV</span><span class="s2">&quot;</span>,if<span class="o">=</span>virtio,format<span class="o">=</span>raw<span class="w"> </span><span class="se">\</span>
<span class="w">    </span>-netdev<span class="w"> </span>tap,id<span class="o">=</span>net0,ifname<span class="o">=</span>tap0,script<span class="o">=</span><span class="s2">&quot;</span><span class="nv">$HOME</span><span class="s2">/bin/qemu-br0&quot;</span>,downscript<span class="o">=</span>no<span class="w"> </span><span class="se">\</span>
<span class="w">    </span>-device<span class="w"> </span>virtio-net-pci,netdev<span class="o">=</span>net0<span class="w"> </span><span class="se">\</span>
<span class="w">    </span>-append<span class="w"> </span><span class="s1">&#39;root=/dev/vda rw&#39;</span>
</pre></div>
</div>
<p>The new relevant bits here are (a) executing the VM in the netns we have
constructed; (b) a <code class="docutils literal notranslate"><span class="pre">-netdev</span></code>  command to configure a tap device; (c) a
virtual network card for the VM. There is also a script <code class="docutils literal notranslate"><span class="pre">$HOME/bin/qemu-br0</span></code>
run by qemu to configure the tap device it creates for the VM:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>#!/bin/bash
tap=$1
ip link set &quot;$tap&quot; master br0
ip link set dev &quot;$tap&quot; up
</pre></div>
</div>
<p>That simply plugs the new tap device into the bridge.</p>
<p>This is all well and good but we are now missing one last crucial step. What is
the IP address of the VM?  There are two options:</p>
<ol class="arabic simple">
<li><p>configure a static IP but the VM’s root device networking stack
configuration must be modified</p></li>
<li><p>use DHCP and configure the root device for VMs to always use dhcp to
configure their ethernet device addresses</p></li>
</ol>
<p>The second option is more complicated to setup, since you must run a DHCP
server now, but provides the greatest flexibility for adding more VMs as needed
when testing.</p>
<p>The modified (or “hacked”) standard dhcpd systemd service looks like:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># cat sepian-dhcpd.service</span>
<span class="p">[</span><span class="n">Unit</span><span class="p">]</span>
<span class="n">Description</span><span class="o">=</span><span class="n">IPv4</span> <span class="n">DHCP</span> <span class="n">server</span>
<span class="n">After</span><span class="o">=</span><span class="n">network</span><span class="o">.</span><span class="n">target</span> <span class="n">network</span><span class="o">-</span><span class="n">online</span><span class="o">.</span><span class="n">target</span> <span class="n">sepian</span><span class="o">-</span><span class="n">netns</span><span class="o">.</span><span class="n">service</span>
<span class="n">Wants</span><span class="o">=</span><span class="n">network</span><span class="o">-</span><span class="n">online</span><span class="o">.</span><span class="n">target</span>
<span class="n">Requires</span><span class="o">=</span><span class="n">sepian</span><span class="o">-</span><span class="n">netns</span><span class="o">.</span><span class="n">service</span>

<span class="p">[</span><span class="n">Service</span><span class="p">]</span>
<span class="n">ExecStartPre</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">touch</span> <span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">dhcpd</span><span class="o">.</span><span class="n">leases</span>
<span class="n">ExecStartPre</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">cat</span> <span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">netns</span><span class="o">/</span><span class="n">sepian</span><span class="o">/</span><span class="n">dhcpd</span><span class="o">.</span><span class="n">conf</span>
<span class="n">ExecStart</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">dhcpd</span> <span class="o">-</span><span class="n">f</span> <span class="o">-</span><span class="mi">4</span> <span class="o">-</span><span class="n">q</span> <span class="o">-</span><span class="n">cf</span> <span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">netns</span><span class="o">/</span><span class="n">sepian</span><span class="o">/</span><span class="n">dhcpd</span><span class="o">.</span><span class="n">conf</span> <span class="o">-</span><span class="n">lf</span> <span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">dhcpd</span><span class="o">.</span><span class="n">leases</span>
<span class="n">NetworkNamespacePath</span><span class="o">=/</span><span class="n">var</span><span class="o">/</span><span class="n">run</span><span class="o">/</span><span class="n">netns</span><span class="o">/</span><span class="n">sepian</span>
<span class="n">RuntimeDirectory</span><span class="o">=</span><span class="n">dhcpd4</span>
<span class="n">User</span><span class="o">=</span><span class="n">dhcp</span>
<span class="n">AmbientCapabilities</span><span class="o">=</span><span class="n">CAP_NET_BIND_SERVICE</span> <span class="n">CAP_NET_RAW</span>
<span class="n">ProtectSystem</span><span class="o">=</span><span class="n">full</span>
<span class="n">ProtectHome</span><span class="o">=</span><span class="n">on</span>
<span class="n">KillSignal</span><span class="o">=</span><span class="n">SIGINT</span>
<span class="c1"># We pull in network-online.target for a configured network connection.</span>
<span class="c1"># However this is not guaranteed to be the network connection our</span>
<span class="c1"># networks are configured for. So try to restart on failure with a delay</span>
<span class="c1"># of two seconds. Rate limiting kicks in after 12 seconds.</span>
<span class="n">RestartSec</span><span class="o">=</span><span class="mi">2</span><span class="n">s</span>
<span class="n">Restart</span><span class="o">=</span><span class="n">on</span><span class="o">-</span><span class="n">failure</span>
<span class="n">StartLimitInterval</span><span class="o">=</span><span class="mi">12</span><span class="n">s</span>

<span class="p">[</span><span class="n">Install</span><span class="p">]</span>
<span class="n">WantedBy</span><span class="o">=</span><span class="n">multi</span><span class="o">-</span><span class="n">user</span><span class="o">.</span><span class="n">target</span>
</pre></div>
</div>
<p>Similarly, the referenced dhcpd.conf:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># cat /etc/netns/sepian/dhcpd.conf</span>
<span class="n">option</span> <span class="n">domain</span><span class="o">-</span><span class="n">name</span><span class="o">-</span><span class="n">servers</span> <span class="mf">192.168.20.2</span><span class="p">;</span>
<span class="n">option</span> <span class="n">subnet</span><span class="o">-</span><span class="n">mask</span> <span class="mf">255.255.255.0</span><span class="p">;</span>
<span class="n">option</span> <span class="n">routers</span> <span class="mf">192.168.0.1</span><span class="p">;</span>
<span class="n">subnet</span> <span class="mf">192.168.0.0</span> <span class="n">netmask</span> <span class="mf">255.255.255.0</span> <span class="p">{</span>
    <span class="nb">range</span> <span class="mf">192.168.0.100</span> <span class="mf">192.168.0.199</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
</div>
<p>Importantly, this tells the VM to route traffic to 192.168.0.1 (the IP of the
bridge in the netns) and DNS can be provided by 192.168.20.2 (via
<code class="docutils literal notranslate"><span class="pre">systemd-resolved</span></code> on the sepia-bounce VM).</p>
<p>In the VM, the networking looks like:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">[</span><span class="n">root</span><span class="nd">@archlinux</span> <span class="o">~</span><span class="p">]</span><span class="c1"># ip link</span>
<span class="mi">1</span><span class="p">:</span> <span class="n">lo</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">LOOPBACK</span><span class="p">,</span><span class="n">UP</span><span class="p">,</span><span class="n">LOWER_UP</span><span class="o">&gt;</span> <span class="n">mtu</span> <span class="mi">65536</span> <span class="n">qdisc</span> <span class="n">noqueue</span> <span class="n">state</span> <span class="n">UNKNOWN</span> <span class="n">mode</span> <span class="n">DEFAULT</span> <span class="n">group</span> <span class="n">default</span> <span class="n">qlen</span> <span class="mi">1000</span>
<span class="n">link</span><span class="o">/</span><span class="n">loopback</span> <span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span> <span class="n">brd</span> <span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span>
<span class="mi">2</span><span class="p">:</span> <span class="n">enp0s3</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">BROADCAST</span><span class="p">,</span><span class="n">MULTICAST</span><span class="p">,</span><span class="n">UP</span><span class="p">,</span><span class="n">LOWER_UP</span><span class="o">&gt;</span> <span class="n">mtu</span> <span class="mi">1500</span> <span class="n">qdisc</span> <span class="n">pfifo_fast</span> <span class="n">state</span> <span class="n">UP</span> <span class="n">mode</span> <span class="n">DEFAULT</span> <span class="n">group</span> <span class="n">default</span> <span class="n">qlen</span> <span class="mi">1000</span>
<span class="n">link</span><span class="o">/</span><span class="n">ether</span> <span class="mi">52</span><span class="p">:</span><span class="mi">54</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">12</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">56</span> <span class="n">brd</span> <span class="n">ff</span><span class="p">:</span><span class="n">ff</span><span class="p">:</span><span class="n">ff</span><span class="p">:</span><span class="n">ff</span><span class="p">:</span><span class="n">ff</span><span class="p">:</span><span class="n">ff</span>
<span class="mi">3</span><span class="p">:</span> <span class="n">sit0</span><span class="nd">@NONE</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">NOARP</span><span class="o">&gt;</span> <span class="n">mtu</span> <span class="mi">1480</span> <span class="n">qdisc</span> <span class="n">noop</span> <span class="n">state</span> <span class="n">DOWN</span> <span class="n">mode</span> <span class="n">DEFAULT</span> <span class="n">group</span> <span class="n">default</span> <span class="n">qlen</span> <span class="mi">1000</span>
<span class="n">link</span><span class="o">/</span><span class="n">sit</span> <span class="mf">0.0.0.0</span> <span class="n">brd</span> <span class="mf">0.0.0.0</span>
<span class="p">[</span><span class="n">root</span><span class="nd">@archlinux</span> <span class="o">~</span><span class="p">]</span><span class="c1"># ip addr</span>
<span class="mi">1</span><span class="p">:</span> <span class="n">lo</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">LOOPBACK</span><span class="p">,</span><span class="n">UP</span><span class="p">,</span><span class="n">LOWER_UP</span><span class="o">&gt;</span> <span class="n">mtu</span> <span class="mi">65536</span> <span class="n">qdisc</span> <span class="n">noqueue</span> <span class="n">state</span> <span class="n">UNKNOWN</span> <span class="n">group</span> <span class="n">default</span> <span class="n">qlen</span> <span class="mi">1000</span>
<span class="n">link</span><span class="o">/</span><span class="n">loopback</span> <span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span> <span class="n">brd</span> <span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span>
<span class="n">inet</span> <span class="mf">127.0.0.1</span><span class="o">/</span><span class="mi">8</span> <span class="n">scope</span> <span class="n">host</span> <span class="n">lo</span>
<span class="n">valid_lft</span> <span class="n">forever</span> <span class="n">preferred_lft</span> <span class="n">forever</span>
<span class="n">inet6</span> <span class="p">::</span><span class="mi">1</span><span class="o">/</span><span class="mi">128</span> <span class="n">scope</span> <span class="n">host</span> <span class="n">noprefixroute</span>
<span class="n">valid_lft</span> <span class="n">forever</span> <span class="n">preferred_lft</span> <span class="n">forever</span>
<span class="mi">2</span><span class="p">:</span> <span class="n">enp0s3</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">BROADCAST</span><span class="p">,</span><span class="n">MULTICAST</span><span class="p">,</span><span class="n">UP</span><span class="p">,</span><span class="n">LOWER_UP</span><span class="o">&gt;</span> <span class="n">mtu</span> <span class="mi">1500</span> <span class="n">qdisc</span> <span class="n">pfifo_fast</span> <span class="n">state</span> <span class="n">UP</span> <span class="n">group</span> <span class="n">default</span> <span class="n">qlen</span> <span class="mi">1000</span>
<span class="n">link</span><span class="o">/</span><span class="n">ether</span> <span class="mi">52</span><span class="p">:</span><span class="mi">54</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">12</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">56</span> <span class="n">brd</span> <span class="n">ff</span><span class="p">:</span><span class="n">ff</span><span class="p">:</span><span class="n">ff</span><span class="p">:</span><span class="n">ff</span><span class="p">:</span><span class="n">ff</span><span class="p">:</span><span class="n">ff</span>
<span class="n">inet</span> <span class="mf">192.168.0.100</span><span class="o">/</span><span class="mi">24</span> <span class="n">metric</span> <span class="mi">1024</span> <span class="n">brd</span> <span class="mf">192.168.0.255</span> <span class="n">scope</span> <span class="k">global</span> <span class="n">dynamic</span> <span class="n">enp0s3</span>
<span class="n">valid_lft</span> <span class="mi">28435</span><span class="n">sec</span> <span class="n">preferred_lft</span> <span class="mi">28435</span><span class="n">sec</span>
<span class="n">inet6</span> <span class="n">fe80</span><span class="p">::</span><span class="mi">5054</span><span class="p">:</span><span class="n">ff</span><span class="p">:</span><span class="n">fe12</span><span class="p">:</span><span class="mi">3456</span><span class="o">/</span><span class="mi">64</span> <span class="n">scope</span> <span class="n">link</span> <span class="n">proto</span> <span class="n">kernel_ll</span>
<span class="n">valid_lft</span> <span class="n">forever</span> <span class="n">preferred_lft</span> <span class="n">forever</span>
<span class="mi">3</span><span class="p">:</span> <span class="n">sit0</span><span class="nd">@NONE</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">NOARP</span><span class="o">&gt;</span> <span class="n">mtu</span> <span class="mi">1480</span> <span class="n">qdisc</span> <span class="n">noop</span> <span class="n">state</span> <span class="n">DOWN</span> <span class="n">group</span> <span class="n">default</span> <span class="n">qlen</span> <span class="mi">1000</span>
<span class="n">link</span><span class="o">/</span><span class="n">sit</span> <span class="mf">0.0.0.0</span> <span class="n">brd</span> <span class="mf">0.0.0.0</span>
<span class="p">[</span><span class="n">root</span><span class="nd">@archlinux</span> <span class="o">~</span><span class="p">]</span><span class="c1"># systemd-resolve --status</span>
<span class="n">Global</span>
        <span class="n">Protocols</span><span class="p">:</span> <span class="o">+</span><span class="n">LLMNR</span> <span class="o">+</span><span class="n">mDNS</span> <span class="o">-</span><span class="n">DNSOverTLS</span> <span class="n">DNSSEC</span><span class="o">=</span><span class="n">no</span><span class="o">/</span><span class="n">unsupported</span>
<span class="n">resolv</span><span class="o">.</span><span class="n">conf</span> <span class="n">mode</span><span class="p">:</span> <span class="n">stub</span>
<span class="n">Fallback</span> <span class="n">DNS</span> <span class="n">Servers</span><span class="p">:</span> <span class="mf">1.1.1.1</span><span class="c1">#cloudflare-dns.com 9.9.9.9#dns.quad9.net 8.8.8.8#dns.google 2606:4700:4700::1111#cloudflare-dns.com 2620:fe::9#dns.quad9.net 2001:4860:4860::8888#dns.google</span>

<span class="n">Link</span> <span class="mi">2</span> <span class="p">(</span><span class="n">enp0s3</span><span class="p">)</span>
<span class="n">Current</span> <span class="n">Scopes</span><span class="p">:</span> <span class="n">DNS</span> <span class="n">LLMNR</span><span class="o">/</span><span class="n">IPv4</span> <span class="n">LLMNR</span><span class="o">/</span><span class="n">IPv6</span>
        <span class="n">Protocols</span><span class="p">:</span> <span class="o">+</span><span class="n">DefaultRoute</span> <span class="o">+</span><span class="n">LLMNR</span> <span class="o">-</span><span class="n">mDNS</span> <span class="o">-</span><span class="n">DNSOverTLS</span> <span class="n">DNSSEC</span><span class="o">=</span><span class="n">no</span><span class="o">/</span><span class="n">unsupported</span>
<span class="n">Current</span> <span class="n">DNS</span> <span class="n">Server</span><span class="p">:</span> <span class="mf">192.168.20.2</span>
<span class="n">DNS</span> <span class="n">Servers</span><span class="p">:</span> <span class="mf">192.168.20.2</span>

<span class="n">Link</span> <span class="mi">3</span> <span class="p">(</span><span class="n">sit0</span><span class="p">)</span>
<span class="n">Current</span> <span class="n">Scopes</span><span class="p">:</span> <span class="n">none</span>
        <span class="n">Protocols</span><span class="p">:</span> <span class="o">-</span><span class="n">DefaultRoute</span> <span class="o">+</span><span class="n">LLMNR</span> <span class="o">+</span><span class="n">mDNS</span> <span class="o">-</span><span class="n">DNSOverTLS</span> <span class="n">DNSSEC</span><span class="o">=</span><span class="n">no</span><span class="o">/</span><span class="n">unsupported</span>
</pre></div>
</div>
<p>Finally, some other networking configurations to consider:</p>
<ul class="simple">
<li><p>Run the VM on your machine with full access to the host networking stack. If you have the sepia vpn, this will probably work without too much configuration.</p></li>
<li><p>Run the VM in a netns as above but also setup the sepia vpn in the same netns. This can help to avoid using a sepia-bounce VM. You’ll still need to configure routing between the bridge and the sepia VPN.</p></li>
<li><p>Run the VM in a netns as above but only use a local vstart cluster (possibly in another VM) in the same netns.</p></li>
</ul>
<section id="step-four-mounting-a-cephfs-file-system-in-your-vm">
<h2>Step Four: mounting a CephFS file system in your VM<a class="headerlink" href="#step-four-mounting-a-cephfs-file-system-in-your-vm" title="Permalink to this heading"></a></h2>
<p>This guide uses a vstart cluster on a machine in the sepia lab. Because the mon
addresses will change with any new vstart cluster, it will invalidate any
static configuration we may setup for our VM mounting the CephFS via the kernel
driver.  So, we should create a script to fetch the configuration for our
vstart cluster prior to mounting:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="ch">#!/bin/bash</span>
<span class="c1"># kmount.sh -- mount a vstart Ceph cluster on a remote machine</span>

<span class="c1"># the cephx client credential, vstart creates &quot;client.fs&quot; by default</span>
<span class="nv">NAME</span><span class="o">=</span>fs
<span class="c1"># static fs name, vstart creates an &quot;a&quot; file system by default</span>
<span class="nv">FS</span><span class="o">=</span>a
<span class="c1"># where to mount on the VM</span>
<span class="nv">MOUNTPOINT</span><span class="o">=</span>/mnt
<span class="c1"># cephfs mount point (root by default)</span>
<span class="nv">CEPHFS_MOUNTPOINT</span><span class="o">=</span>/

<span class="k">function</span><span class="w"> </span>run<span class="w"> </span><span class="o">{</span>
<span class="w">    </span><span class="nb">printf</span><span class="w"> </span><span class="s1">&#39;%s\n&#39;</span><span class="w"> </span><span class="s2">&quot;</span><span class="nv">$*</span><span class="s2">&quot;</span><span class="w"> </span>&gt;<span class="p">&amp;</span><span class="m">2</span>
<span class="w">    </span><span class="s2">&quot;</span><span class="nv">$@</span><span class="s2">&quot;</span>
<span class="o">}</span>

<span class="k">function</span><span class="w"> </span>mssh<span class="w"> </span><span class="o">{</span>
<span class="w">    </span>run<span class="w"> </span>ssh<span class="w"> </span>vossi04.front.sepia.ceph.com<span class="w"> </span><span class="s2">&quot;cd ceph/build &amp;&amp; (source vstart_environment.sh; </span><span class="nv">$1</span><span class="s2">)&quot;</span>
<span class="o">}</span>

<span class="c1"># create the minimum config (including mon addresses) and store it in the VM&#39;s ceph.conf. This is not used for mounting; we&#39;re storing it for potential use with `ceph` commands.</span>
mssh<span class="w"> </span><span class="s2">&quot;ceph config generate-minimal-conf&quot;</span><span class="w"> </span>&gt;<span class="w"> </span>/etc/ceph/ceph.conf
<span class="c1"># get the vstart cluster&#39;s fsid</span>
<span class="nv">FSID</span><span class="o">=</span><span class="k">$(</span>mssh<span class="w"> </span><span class="s2">&quot;ceph fsid&quot;</span><span class="k">)</span>
<span class="c1"># get the auth key associated with client.fs</span>
<span class="nv">KEY</span><span class="o">=</span><span class="k">$(</span>mssh<span class="w"> </span><span class="s2">&quot;ceph auth get-key client.</span><span class="nv">$NAME</span><span class="s2">&quot;</span><span class="k">)</span>
<span class="c1"># dump the v2 mon addresses and format for the -o mon_addr mount option</span>
<span class="nv">MONS</span><span class="o">=</span><span class="k">$(</span>mssh<span class="w"> </span><span class="s2">&quot;ceph mon dump --format=json&quot;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>-r<span class="w"> </span><span class="s1">&#39;.mons[] | .public_addrs.addrvec[] | select(.type == &quot;v2&quot;) | .addr&#39;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>paste<span class="w"> </span>-s<span class="w"> </span>-d/<span class="k">)</span>

<span class="c1"># turn on kernel debugging (and any other debugging you&#39;d like)</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">&quot;module ceph +p&quot;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>tee<span class="w"> </span>/sys/kernel/debug/dynamic_debug/control
<span class="c1"># do the mount! we use the new device syntax for this mount</span>
run<span class="w"> </span>mount<span class="w"> </span>-t<span class="w"> </span>ceph<span class="w"> </span><span class="s2">&quot;</span><span class="si">${</span><span class="nv">NAME</span><span class="si">}</span><span class="s2">@</span><span class="si">${</span><span class="nv">FSID</span><span class="si">}</span><span class="s2">.</span><span class="si">${</span><span class="nv">FS</span><span class="si">}</span><span class="s2">=</span><span class="si">${</span><span class="nv">CEPHFS_MOUNTPOINT</span><span class="si">}</span><span class="s2">&quot;</span><span class="w"> </span>-o<span class="w"> </span><span class="s2">&quot;mon_addr=</span><span class="si">${</span><span class="nv">MONS</span><span class="si">}</span><span class="s2">,ms_mode=crc,name=</span><span class="si">${</span><span class="nv">NAME</span><span class="si">}</span><span class="s2">,secret=</span><span class="si">${</span><span class="nv">KEY</span><span class="si">}</span><span class="s2">,norequire_active_mds,noshare&quot;</span><span class="w"> </span><span class="s2">&quot;</span><span class="nv">$MOUNTPOINT</span><span class="s2">&quot;</span>
</pre></div>
</div>
<p>That would be run like:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$<span class="w"> </span>sudo<span class="w"> </span>ip<span class="w"> </span>netns<span class="w"> </span><span class="nb">exec</span><span class="w"> </span>sepian<span class="w"> </span>ssh<span class="w"> </span>root@192.168.0.100<span class="w"> </span>./kmount.sh
...
mount<span class="w"> </span>-t<span class="w"> </span>ceph<span class="w"> </span>fs@c9653bca-110b-4f70-9f84-5a195b205e9a.a<span class="o">=</span>/<span class="w"> </span>-o<span class="w"> </span><span class="nv">mon_addr</span><span class="o">=</span><span class="m">172</span>.21.10.4:40762/172.21.10.4:40764/172.21.10.4:40766,ms_mode<span class="o">=</span>crc,name<span class="o">=</span>fs,secret<span class="o">=</span><span class="nv">AQD0jgln43pBCxAA7cJlZ4Px7J0UmiK4A4j3rA</span><span class="o">==</span>,norequire_active_mds,noshare<span class="w"> </span>/mnt
$<span class="w"> </span>sudo<span class="w"> </span>ip<span class="w"> </span>netns<span class="w"> </span><span class="nb">exec</span><span class="w"> </span>sepian<span class="w"> </span>ssh<span class="w"> </span>root@192.168.0.100<span class="w"> </span>df<span class="w"> </span>-h<span class="w"> </span>/mnt
Filesystem<span class="w">                                   </span>Size<span class="w">  </span>Used<span class="w"> </span>Avail<span class="w"> </span>Use%<span class="w"> </span>Mounted<span class="w"> </span>on
fs@c9653bca-110b-4f70-9f84-5a195b205e9a.a<span class="o">=</span>/<span class="w">  </span>169G<span class="w">     </span><span class="m">0</span><span class="w">  </span>169G<span class="w">   </span><span class="m">0</span>%<span class="w"> </span>/mnt
</pre></div>
</div>
<p>If you run into difficulties, it may be:</p>
<ul class="simple">
<li><p>The firewall on the node running the vstart cluster is blocking your connections.</p></li>
<li><p>Some misconfiguration in your networking stack.</p></li>
<li><p>An incorrect configuration for the mount.</p></li>
</ul>
</section>
<section id="step-five-testing-kernel-changes-in-teuthology">
<h2>Step Five: testing kernel changes in teuthology<a class="headerlink" href="#step-five-testing-kernel-changes-in-teuthology" title="Permalink to this heading"></a></h2>
<p>There 3 static branches in the <a class="reference external" href="https://github.com/ceph/ceph-client">ceph kernel git repository</a> managed by the Ceph team:</p>
<ul class="simple">
<li><p><a class="reference external" href="https://github.com/ceph/ceph-client/tree/for-linus">for-linus</a>: A branch managed by the primary Ceph maintainer to share changes with Linus Torvalds (upstream). Do not push to this branch.</p></li>
<li><p><a class="reference external" href="https://github.com/ceph/ceph-client/tree/master">master</a>: A staging ground for patches planned to be sent to Linus. Do not push to this branch.</p></li>
<li><p><a class="reference external" href="https://github.com/ceph/ceph-client/tree/testing">testing</a> A staging ground for miscellaneous patches that need wider QA testing (via nightlies or regular Ceph QA testing). Push patches you believe to be nearly ready for upstream acceptance.</p></li>
</ul>
<p>You may also push a <code class="docutils literal notranslate"><span class="pre">wip-$feature</span></code> branch to the <code class="docutils literal notranslate"><span class="pre">ceph-client.git</span></code>
repository which will be built by Jenkins. Then view the results of the build
in <a class="reference external" href="https://shaman.ceph.com/builds/kernel/">Shaman</a>.</p>
<p>Once a kernel branch is built, you can test it via the <code class="docutils literal notranslate"><span class="pre">fs</span></code> CephFS QA suite:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$<span class="w"> </span>teuthology-suite<span class="w"> </span>...<span class="w"> </span>--suite<span class="w"> </span>fs<span class="w"> </span>--kernel<span class="w"> </span>wip-<span class="nv">$feature</span><span class="w"> </span>--filter<span class="w"> </span>k-testing
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">k-testing</span></code> filter is looking for the fragment which normally sets
<code class="docutils literal notranslate"><span class="pre">testing</span></code> branch of the kernel for routine QA. That is, the <code class="docutils literal notranslate"><span class="pre">fs</span></code> suite
regularly runs tests against whatever is in the <code class="docutils literal notranslate"><span class="pre">testing</span></code> branch of the
kernel. We are overriding that choice of kernel branch via the <code class="docutils literal notranslate"><span class="pre">--kernel</span>
<span class="pre">wip-$featuree</span></code> switch.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Without filtering for <code class="docutils literal notranslate"><span class="pre">k-testing</span></code>, the <code class="docutils literal notranslate"><span class="pre">fs</span></code> suite will also run jobs using ceph-fuse or stock kernel, libcephfs tests, and other tests that may not be of interest to you when evaluating changes to the kernel.</p>
</div>
<p>The actual override is controlled using Lua merge scripts in the
<code class="docutils literal notranslate"><span class="pre">k-testing.yaml</span></code> fragment. See that file for more details.</p>
</section>
</section>



<div id="support-the-ceph-foundation" class="admonition note">
  <p class="first admonition-title">Brought to you by the Ceph Foundation</p>
  <p class="last">The Ceph Documentation is a community resource funded and hosted by the non-profit <a href="https://ceph.io/en/foundation/">Ceph Foundation</a>. If you would like to support this and our other efforts, please consider <a href="https://ceph.io/en/foundation/join/">joining now</a>.</p>
</div>


           </div>
           
          </div>
          <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
        <a href="../iana/" class="btn btn-neutral float-left" title="IANA 号" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
        <a href="../kubernetes/" class="btn btn-neutral float-right" title="Hacking on Ceph in Kubernetes with Rook" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>&#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).</p>
  </div>

   

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>